Expert Political Judgment: How Good Is It? How Can We Know? by Philip Tetlock is extraordinarily important. I hesitate to call a book "essential reading" for a thoughtful person, but Tetlock qualifies if anything does. After a rigorous multi-year study of hundreds of experts and a detailed analysis of their responses to diverse questions about future political and economic developments, the Big Conclusion is that expert judgment is quite bad, only slightly better than random chance and significantly worse than formal models. People, even those who should know better, have unconscious bias. They're inconsistent and illogical. They explain away or forget their failures and they exaggerate their successes. Experts are no exception.

But the Good News is that some experts are significantly better than others. As Tetlock puts it:

... What experts think matters far less than how they think. If we want realistic odds on what will happen next, coupled to a willingness to admit mistakes, we are better off turning to experts who embody the intellectual traits of Isaiah Berlin's prototypical fox—those who "know many little things," draw from an eclectic array of traditions, and accept ambiguity and contradiction as inevitable features of life—than we are turning to Berlin's hedgehogs—those who "know one big thing," toil devotedly within one tradition, and reach for fomulaic solutions to ill-defined problems. ...

Expert Political Judgment is full of statistical analyses. Those who don't enjoy conditional (Bayesian) probabilities and correlation coefficients may wish to read it like a novel, skip the equations, and focus on the graphs and associated discussion. Throughout Tetlock is scrupulously fair: he bends over backwards to understand, allow for, and explain potential flaws in his research. An entire chapter—"The Hedgehogs Strike Back"—is devoted to making the opposition's case. His study of scenarios highlights the rarely-recognized risks of using them. And throughout the book, Tetlock's dry humor sparkles.

Bottom line: I wish I had read Expert Political Judgment a few dozen years ago.

^z - 2010-05-13